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AIRLINE QUALITY RATING 2004 


Brent D. Bowen, University of Nebraska at Omaha 
Dean E. Headley, Wichita State University 

Abstract 

The Airline Quality Rating (AQR) was developed and first announced in early 
1991 as an objective method of comparing airline quality on combined multiple 
performance criteria. This current report, the Airline Quality Rating 2004, reflects 
monthly Airline Quality Rating scores for 2003. AQR scores for the calendar year 
2003 are based on 15 elements in four major areas that focus on airline performance 
aspects important to air travel consumers. 

The Airline Quality Rating 2004 is a summary of month-by-month quality 
ratings for U.S. airlines that have at least 1% of domestic passenger volume during 
2003. Using the Airline Quality Rating system of weighted averages and monthly 
performance data in the areas of on-time arrivals, involuntary denied boardings, 
mishandled baggage, and a combination of 12 customer complaint categories, 
airlines’ comparative performance for the calendar year of 2003 is reported. This 
research monograph contains a brief summary of the AQR methodology, detailed 
data and charts that track comparative quality for domestic airline operations for the 
12-month period of 2003, and industry results. Also, comparative Airline Quality 
Rating data for 2002 are included, where available, to provide historical perspective 
regarding performance quality in the industry. 


The Airline Quality Rating (AQR) System 

The majority of quality ratings available rely on subjective surveys of consumer 
opinion that are infrequently done. This subjective approach yields a quality rating 
that is essentially non-comparable from survey to survey for any specific airline. 
Timeliness of survey-based results can be a problem in the fast-paced airline industry 
as well. Before the Airline Quality Rating, there was effectively no consistent method 
for monitoring the quality of airlines on a timely, objective, and comparable basis. 

With the introduction of the AQR, a multi-factor, weighted average approach became 
available that had not been used before in the airline industry. The method relies on 
taking published, publicly available data that reports actual airline performance on 
critical quality criteria important to consumers and combines them into a rating 
system. The final result is a rating for individual airlines with interval scale properties 
that is comparable across airlines and across time. 

The Airline Quality Rating (AQR) is a weighted average of multiple elements 
(see Table 1) important to consumers when judging the quality of airline services. 
Elements considered for inclusion in the rating scale were screened to meet two 
basic criteria; 1) an element must be obtainable from published data sources for each 
airline; and 2) an element must have relevance to consumer concerns regarding 
airline quality. Data for the elements used in calculating the ratings represent 
performance aspects (on-time arrival, mishandled baggage, involuntary denied 
boardings, and 12 customer complaint areas) of airlines that are important to 


consumers. All of the elements are reported in the Air Travel Consumer Report 
maintained by the U.S. Department of Transportation. 

Weights were originally established by surveying 65 airline industry experts 
regarding their opinion as to what consumers would rate as important (on a scale of 0 
to 10) in judging airline quality. Also, each weight and element was assigned a plus 
or minus sign to reflect the nature of impact for that criterion on a consumer’s 
perception of quality. For instance, the criteria of on-time arrival performance are 
included as a positive element because it is reported in terms of on-time successes, 
suggesting that a higher number is favorable to consumers. The weight for this 
criterion is high due to the importance most consumers place on this aspect of airline 
service. Conversely, the criteria that includes mishandled baggage is included as a 
negative element because it is reported in terms of mishandled bags per 1000 
passengers served, suggesting that a higher number is unfavorable to consumers. 
Because having baggage arrive with passengers is important to consumers the 
weight for this criterion is also high. Weights and positive/negative signs are 
independent of each other. Weights reflect importance of the criteria in consumer 
decision-making, while signs reflect the direction of impact that the criteria should 
have on the consumer's rating of airline quality. When all criteria, weights and 
impacts are combined for an airline over the year, a single interval scaled value is 
obtained. This value is comparable across airlines and across time periods. In the 
spring of 2002, a nationwide survey of frequent flyers was conducted that allowed a 
revisiting of the weighting for the AQR elements. Analysis of the sample of 766 
opinions showed no appreciable difference in the relative weights for the AQR 
elements. To maintain comparability across the years, the weights have been kept 
the same as originally established. 

The Airline Quality Rating criteria and the weighted average methodology 
allow a focused comparison of airline domestic performance. Unlike other consumer 
opinion approaches that rely on consumer surveys and subjective opinion, the AQR 
continues to use a mathematical formula that takes multiple weighted objective 
criteria into account in arriving at a single, fully comparable rating for airline industry 
performance. The Airline Quality Rating provides both consumers and industry 
watchers a means for looking at comparative quality for each airline on a timely 
basis, using objective, performance-based data. Over the years, the Airline Quality 
Rating has often been cited as an industry standard for comparing airline 
performance. With the continued global trend in airline operations alliances, the 
argument becomes even stronger for the Airline Quality Rating to be used as a 
standard method for comparing the quality of airline performance for international 
operations as well. 


Table 1 


AIRLINE QUALITY RATING CRITERIA, WEIGHTS AND IMPACT 

CRITERIA WEIGHT IMPACT (+/-) 


OT 

On-Time 

8.63 

+ 

DB 

Denied Boardings 

8.03 

— 

MB 

Mishandled Baggage 

7.92 

— 

CC 

Customer Complaints 
Flight Problems 

7.17 

— 


Oversales 

Reservations, Ticketing, and Boarding 

Fares 

Refunds 

Baggage 

Customer Service 

Disability 

Advertising 

Discrimination 

Animals 

Other 

Data for all criteria is drawn from the U.S. Department of Transportation’s monthly 
Air Travel Consumer Report, (http://dot.gov/airconsumer/) 


The formula for calculating the AQR score is: 

(+8.63 x OT) + (-8.03 x DB) + (-7.92 x MB) + (-7.17 x CC) 


AQR = 


(8.63 + 8.03 + 7.92 + 7.17) 


What the Airline Quality Rating Tells Us About 2003 

The Airline Quality Rating industry score shows an industry that is improving in 
quality relative to customer performance criteria. Of the ten carriers rated in both 2002 
and 2003, only American Airlines and US Airways, show declines in their overall AQR 
scores for 2003. The AQR score for Delta Airlines in 2003 was virtually unchanged from 
their 2002 level. Northwest Airlines registered the largest improvement in AQR score 
over the past year. Four new carriers are included in the AQR (a total of 14) for 2003. 
AQR results for 2003 indicate that: 

For 2003 the overall industry AQR score was slightly better than in 2002 despite 
decreased industry performance in three of the four areas. As an industry, the AQR 
criteria shows that on-time arrival percentage was down slightly (82.0% in 2003 
compared to 82.1% in 2002), involuntary denied boardings per passenger served 
increased (0.72 per 10,000 passengers in 2002 to 0.86 per 10,000 passengers in 2003), 
mishandled baggage rates increased (3.84 per 1 ,000 passengers in 2002 versus 4.00 
per 1 ,000 passengers in 2003), and consumer complaint rates improved (1 .22 per 

100.000 passengers in 2002 down to 0.67 per 100,000 passengers in 2003). Taken 
together, the AQR score for the industry improved from a level of -1 .19 in 2002 to -1 .14 in 
2003. With three of the four rating categories (On-Time, Denied Boardings, and 
Mishandled Baggage) having poorer performance in 2003 than in 2002, the dramatic 
improvement in Customer Complaint rates (nearly a 50% reduction) is the reason for the 
small overall industry AQR score improvement. 

AirTran Airlines (FL) is included in the AQR for the first time in 2003. On-time 
performance was among the lowest (78.1%) of the airlines rated. AirTran’s denied 
boardings performance (1 .45 per 10,000 passengers compared to an industry average of 
0.86) was among the highest of the airlines rated. A customer complaint rate of 0.83 
complaints per 100,000 passengers was also above the industry average rate of 0.67. 
The positive news is that their mishandled baggage rate of 2.84 per 1 ,000 passengers 
(second best to Alaska Airlines) is well below the industry average rate of 4.00 bags per 

1.000 passengers. 

Alaska Airlines (AS) had an improved AQR score for 2003. Consistent solid 
performance in the areas of on-time (78.0% in 2002 compared to 81 .0% in 2003), 
involuntary denied boardings (1.17 per 10,000 passengers in 2002 compared to 0.81 in 
2003), customer complaints (0.91 per 100,000 passengers in 2002 compared to 0.52 in 
2003), and an industry best baggage handling rate (2.56 mishandled bags per 1,000 
passengers in 2003 compared to 2.63 in 2002) helped Alaska Airlines stay near the top 
of the ratings. 

America West Airlines (HP) showed improvement in their AQR score for 2003. On-time 
performance slipped slightly in 2003 (82.9% in 2002 to 82.0% in 2003). The rate of 
mishandled baggage improved from 3.55 in 2002 to 3.30 in 2003. Consumer complaints 
were reduced by nearly 50%, (1.63 in 2002 to 0.84 in 2003) for the third year in a row. 
Denied boarding rates increased, moving from 0.20 per 10,000 passengers served in 
2002 to 0.40 in 2003. 


American Airlines’ (AA) AQR score for 2003 dropped slightly (-1 .21 in 2002 to -1 .24 in 
2003). Their drop in AQR score reflects a decline in performance for on-time arrivals 
(81.7% in 2003 compared to 83.8% in 2002), mishandled baggage rates (4.27 in 2002 
compared to 4.45 in 2003), and denied boardings rates (0.31 in 2002 compared to 0.59 in 
2003). A 30% reduction in the rate of customer complaints (1 .29 in 2002 to 0.88 in 2003) 
was not enough to overcome die combined negative effect of the other elements 
considered in the AQR. 

American Eagle Airlines (MQ) had a denied boardings rate (0.38 for 2003 up from 0.19 
per 10,000 passengers in 2002) that was among the industry best. The airline showed 
improvements in the rate of customer complaints (0.51 in 2003 compared to 0.60 per 

100.000 passengers in 2002). On-time performance was 78.6% in 2003 compared to 
79.1 % for 2002. Their mishandled baggage rate was more than double the industry rate, 
but did show improvement for the year (8.42 1 ,000 passengers in 2003 compared to 9.81 
per 1 ,000 passengers in 2002). This combination of gains and losses made American 
Eagle the second most improved airline for 2003. 

ATA Airlines (TZ) is included in the AQR for the first time in 2003. On-time performance 
for the year, 80.0%, was near the industry average of 81 .3%. ATA’s denied boardings 
performance, 0.89 per 10,000 passengers, compared favorable to an industry average of 
0.86 per 10,000 passengers. A customer complaint rate of 0.66 complaints per 100,000 
passengers was also very dose to the industry average rate of 0.67. Their mishandled 
baggage rate of 4.06 per 1 ,000 passengers is consistent with the industry average rate of 

4.00 bags per 1,000 passengers as well. Overall, ATA performed in all areas rated at a 
level that is consistent with industry averages for 2003. 

Atlantic Southeast Airlines (EV) is included in the AQR for the first time in 2003. On- 
time performance (75.4%) was the lowest of the airlines rated. Atlantic Southeast’s 
denied boardings performance (7.86 per 10,000 passengers compared to an industry 
average of 0.86) was the worst (by a factor of 9 times) of the 14 airlines rated. Their 
mishandled baggage rate of 15.41 per 1,000 passengers is nearly four times the industry 
average rate of 4.00 bags per 1,000 passengers. The only bright spot for Atlantic 
Southeast is that their customer complaint rate of 0.59 complaints per 100,000 
passengers was better than the industry average rate of 0.67 per 100,000 passengers. 

Continental Airlines (CO) posted improved performance in two of the four AQR criteria. 
Better performance in mishandled baggage (3.11 in 2003 versus 3.14 in 2002) and 
consumer complaint rates (0.95 in 2003 compared to 1.37 in 2002) helped Continental. 
Increases in denied boardings (0.87 in 2002 to 1 .06 in 2003) and poorer on-time 
performance (82.0% in 2003 compared to 83.5% in 2002) was enough to make their 
AQR score only slightly better in 2003 (-1 .06 in 2003 versus -1.10 in 2002). 

Delta Airlines (DL) AQR score for 2003 reflects improvement in on-time arrival 
percentage (82.3% in 2003 compared to 80.0% in 2002) and customer complaint rate 
(1.37 in 2002 compared to 0.78 in 2003). The negatives for Delta were a 17% increase 
in the rate of denied boardings (2002 rate of 1.1 1 compared to 2003 rate of 1.30) and an 
increase in the rate of mishandled baggage (3.57 in 2002 to 3.84 in 2003). This 
combination of improvements and decreases in performance combined to give Delta only 
a slight improvement (-1 .26 in 2002 to -1 .24 in 2003) in overall AQR score. 


Jet Blue Airlines (B6) is included in the AQR for the first time in 2003. On-time 
performance (84.3%) was second best among the 14 airiines rated. Jet Biue’s denied 
boardings performance (0.00 per 10,000 passengers) was the lowest of the airlines rated. 
A customer complaint rate of 0.31 complaints per 100,000 passengers was also the 
second best (to Southwest) of all airlines rated. Their mishandled baggage rate of 3.21 
per 1 ,000 passengers is below the industry average rate of 4.00 bags per 1 ,000 
passengers. 

Northwest Airiines (NW) was file most improved airline in 2003. Their AQR score 
dropped from -1 .39 in 2002 to -1 .02 in 2003. Northwest posted improvements in three of 
the four areas of the AQR for 2003. The rate of mishandled baggage decreased from 
4.52 per 1 ,000 passengers in 2002 to 3.42 per 1 ,000 passengers in 2003. On-time 
arrival performance moved from 80.8% in 2002 to 82.9% in 2003. An improvement in 
customer complaint rate from 1 .45 per 100,000 passengers in 2002 to 0.95 per 100,000 
passengers in 2003 was consistent with others in the industry. Denied boardings 
increased from 0.60 per 10,000 passengers in 2002 to 0.70 per 10,000 passengers in 
2003. 

Southwest Airlines (WN) recorded improvement in on-time arrival percentage (86.3% in 
2003 from 82.6% in 2002), involuntary denied boarding rates (1.02 per 10,000 
passengers in 2003 from 1.09 per 10,000 passengers in 2002), mishandled baggage 
rates (3.35 per 1,000 passengers in 2003 from 3.52 per 1,000 passengers in 2002), and 
customer complaint rates (0.33 per 100,000 passengers in 2002 compared to an industry 
best of 0.14 per 100,000 passengers in 2003). Southwest Airlines is consistently the 
airline with the lowest customer complaint rate in the industry (0.14 per 100,000 
passengers in 2003 compared to an industry rate of 0.67 per 100,000 passengers). 

United Airlines (UA) slipped in on-time arrival performance (from 84.0% in 2002 to 
83.3% in 2003) and in mishandled baggage (3.76 per 1,000 passengers in 2002 
compared to 3.93 in 2003). Performance regarding denied boardings (0.65 per 10,000 
passengers in 2003 compared to 0.69 in 2002), and consumer complaints (0.83 in 2003 
compared to 1.71 per 100,000 passengers in 2002) were improved for 2003. The rate of 
consumer complaints was reduced by half in 2003. 

US Airways (US) showed noticeable improvement in only one of the four criteria tracked 
for 2003. Closer inspection reveals that US Airways performed worse in on-time arrival 
percentage (79.7% in 2003 compared to 83.4% in 2002) and mishandled baggage (3.55 
per 1 ,000 passengers in 2003 compared to 2.95 in 2002). Denied boardings for 2003 
(0.34 per 10,000 passengers) were virtually unchanged from 2002 (0.35 per 10,000 
passengers). As with all other airlines rated, the customer complaint rate dropped (0.90 
per 100,000 passengers in 2003 compared to 1.13 in 2002) for US Airways during 2003. 
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Detail of Airline Performance 


Since the Airline Quality Rating is comparable across airlines and across time, monthly 
rating results can be examined both individually and collectively. The following pages 
outline the AQR scores for the industry and for each airline by month for 2003. For 
comparison purposes, results are also displayed for 2002 where available. A composite 
industry chart that combines the airlines tracked is shown at first, with individual airline 
performance charts foliowing in alphabetical order. 
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Detail of Frequently Cited Airline Performance Criteria 


Consumer interest remains high regarding such issues as on-time performance, 
mishandled baggage, involuntary denied boardings (bumping), and treatment of 
customers. Since these criteria are central to the AQR calculations, it is important to 
provide more complete data for individual airlines in these areas. The following data 
tables and charts provide a detailed look at the performance of each of the 14 U.S. 
airlines that handled at least 1% or more of the total passenger volume for 2003 in the 
specific areas of on-time arrivals, mishandled baggage, involuntary denied boardings, 
and consumer complaints. Data were drawn from the U.S. Department of Transportation 
monthly Air Travel Consumer Report. The final pages of this report outiine the Airline 
Quality Rating criteria definitions for reference and clarity in more fully understanding the 
nature of the data reported. 


2003 On-Time Arrival Percentage by Month for U.S. Airlines 


CQ 

3T-oos(ooTfOcoroc3)rocos 

rCOT-CMr-OOOlX5CMCM^CN©COO) 

C KCOCOOOSCONCOOOQOOOGOOOS 


©Ot\JO)^CONO)CO^T-COCOUDtX) 

QSSSStONSNSCONCONS 


>00)OCMr-KCDSCDCOCOlOS(0 

OCNMOCNSr-^(OCOCOCOO)inTf^ 

^COSCONNSCONCOCONOONCD 

*.CO0rMCMOCDO>CM^^CMS^O) 

O^OTj'COlOCOCMCOinOOOCOCDCD 

OopoqcqcooocqoooqajojaocqoqS 

Q.C\JC\J^OincOCD^-T“T-COSO)^- 

d)U)CO0lOCMOOr-cOCOCMCOO)SCM 

w cooqoqoqcqcacqscoo)cocococo 

0)00 COOOIOOCDt-cONCO(NOONN 
300NO^NOCOO)SCOt-iOt-t* 
^<ONOOSSOO(DSNSCOCOCON 

U3CNOS(D0r-Tj-COinNOSS 

^SOOOOSSNtDCOSCOOOCOOON 

3 NT-U)Ot-inCMt-CN4fOlO^S 

^r^oocooqoqoqr^cooooqoqoooON- 

>*0 ^(jStCOO)ScO^COT-tS 
w OSgJJOJCOgCOOlOOlOO 
5wCCC0CCC0C0NCCC0C5C0C5XC0 


V.r-fOCN)OfOWin^tf)OlOCOO)0 
acos ooton^f oocooj s coo st 
^cqcqoooqoqoor^oqoooooqa^oooo 

L^r-inOiT-MOMOOSOOtn 

WCDCNlOlCON^t-NCOtOr-S^CN 

JNCOSCqoONSCOOONCOCOOOCX) 


-Qi;2S^S co S^S2S ocoino,, “ co ^ 

Qp©(\ip)r-00)NST-C00 r- 
ITSSNSSSCOSSNSOOCOK 


pOtOCOOCOOM-OJCMOJinOT- 

-JCOOONOONSNCOCONOOOOCOOO 


O u 

£ S. « 

5. ® So 

■s < W 

5 c c ^(o ■£ 3"'-' 

'COCO (0 r « mo © 

uooP^ c°| 

® ® Sc ™ ffl; 

E E EPS o ® * 


E| 

■o S 
5< 

C (0 
3 3 
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2003 Involuntary Denied Boardings by Quarter for U.S. Airlines 

(per 10,000 passengers) 
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of the Industry value. Due to size of aircraft operated, these carriers are not required to report this statistic. 

Source: Air Travel Consumer Report. U.S. Department of Transportation. Office of Aviation Enforcement and Proceedings. 


2002 Involuntary Denied Boardings by Quarter for U.S. Airlines 
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2003 Mishandled Baggage by Month for U.S. Airlines 

(per 1 ,000 passengers) 
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Airline Quality Rating Criteria Overview 

The individual criteria used to calculate the AQR scores are summed up in four basic 
areas that reflect customer-oriented areas of airline performance. Definitions of the four 
areas used in this AQR 2004 (2003 data) are outlined below. 

OT ON-TIME PERFORMANCE (+8.63) 

Regularly published data regarding on-time arrival performance is obtained from the U.S. 
Department of Transportation's Air Travel Consumer Report. According to the DOT, a 
flight is counted "on time" if it is operated within 15 minutes of the scheduled time shown 
in the carriers' Computerized Reservations Systems. Delays caused by mechanical 
problems are counted as of January 1, 1995. Canceled and diverted operations are 
counted as late. The AQR calculations use the percentage of flights arriving on time for 
each airline for each month. 

DB INVOLUNTARY DENIED BOARDINGS (-8.03) 

This criterion includes involuntary denied boardings. Data regarding denied boardings 
can be obtained from the U.S. Department of Transportation's Air Travel Consumer 
Report. Data includes the number of passengers who hold confirmed reservations and 
are involuntarily denied boarding on a flight that is oversold. These figures include only 
passengers whose oversold flight departs without them onboard. The AQR uses the ratio 
of involuntary denied boardings per 10,000 passengers boarded by month. 

MB MISHANDLED BAGGAGE REPORTS (-7.92) 

Regularly published data regarding consumer reports to the carriers of mishandled 
baggage can be obtained from the U.S. Department of Transportation's Air Travel 
Consumer Report. According to the DOT, a mishandled bag includes claims for lost, 
damaged, delayed, or pilfered baggage. Data Is reported by carriers as to the rate of 
mishandled baggage reports per 1 ,000 passengers and for the industry. The AQR ratio 
is based on the total number of reports each carrier received from passengers 
concerning lost, damaged, delayed, or pilfered baggage per 1,000 passengers served. 

CC CONSUMER COMPLAINTS (-7.17) 

The criteria of consumer complaints is made up of 12 specific complaint categories 
(outlined below) monitored by the U. S. Department of Transportation and reported 
monthly in the Air Travel Consumer Report. Consumers can file complaints with the DOT 
in writing, by telephone, via e-mail, or in person. The AQR uses complaints about the 
various categories as part of the larger customer complaint criteria and calculates the 
consumer complaint ratio on the number of complaints received per 100,000 passengers 
flown for each airline. 


CONSUMER COMPLAINT CATEGORIES 


Flight Problems 

Data is available by the total number of consumer complaints pertaining to cancellations, 
delays, or any other deviations from schedule, whether planned or unplanned for each 
airline each month. 

Oversales 

This complaint category includes all bumping problems, whether or not the airline 
complied with DOT oversale regulations. Data is available by the total number of 
consumer complaints pertaining to oversales for each airline each month. 

Reservations, Ticketing, and Boarding 

This category includes airline or travel agent mistakes in reservations and ticketing, 
problems in making reservations and obtaining tickets due to busy telephone lines, or 
waiting in line or delays in mailing tickets, and problems boarding the aircraft (except 
oversales). Data is available by the total number of consumer complaints pertaining to 
ticketing and boarding for each airline each month. 

Fares 

As defined by the DOT, consumer complaints regarding fares include incorrect or 
incomplete information about fares, discount fare conditions and availability, overcharges, 
fare increases, and level of fares in general. Data is available for the total number of 
consumer complaints pertaining to fares for each airline each month. 

Refunds 

This category includes customer complaints about problems in obtaining refunds for 
unused or lost tickets, fare adjustments, or bankruptcies. Data is available by the total 
number of consumer complaints pertaining to refunds for each airline each month. 

Baggage 

Claims for lost, damaged, or delayed baggage, charges for excess baggage, carry-on 
problems, and difficulties with airline claim procedure are included in this category. Data 
is available by the total number of consumer complaints pertaining to baggage for each 
airline each month. 

Customer Service 

This category includes complaints about rude or unhelpful employees, inadequate meals 
or cabin service, and treatment of delayed passengers. Data is available by the total 
number of consumer complaints pertaining to customer service for each airline each 
month. 


Disability 

Previously included as part of the Reservations, Ticketing and Boarding Category 
(through 6/99), this category includes complaints about civil rights complaints by air 
travelers with disabilities. Data is available by the total number of consumer complaints 
pertaining to disabilities for each airline each month. 

Advertising 

These are complaints concerning advertising that is deemed unfair, misleading or 
offensive to consumers. Data is available by the total number of consumer complaints 
regarding advertising for each airline each month. 

Discrimination 

Civil rights complaints by air travelers (other than disabilities); for example; complaints 
based on race, national origin, religion, etc. (this category was first reported in May, 
2002 ). 

Animals 

This category, added in October 2000, tracks customer complaints about loss, injury, or 
death of an animal during air transport by an air carrier. Data is available by the total 
number of customer complaints regarding animals for each airline each month. 

Other 

Data regarding consumer complaints about frequent flyer programs, smoking, tours 
credit, cargo problems, security, airport facilities, claims for bodily injury, and other 
problems not classified above are included in this category. Data is available by the total 
number of consumer complaints regarding other problems for each airline each month. 


